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Executive Sumidiiry 

The goal of this study y^as the development and validation of a set of 
measures that could be usen to assess the effectiveness of the change from an 
industrial arts teacher education program to technology teacher education. 
Spedflcalty two research questions were asked. The first question asks directly: Is 
the program technology teacher education? That !s, has the change progressed to 
thv point where the program, taken as a whole, meets the criteria for inclusion in 
the category of technology teacher education, rather than industrial arts teacher 
education, industrial teacner education, vocational teacher education, or technical 
teacher education? The second question is more general: To what degree has the 
program moved effectively to accomplish the change to technology teacher education 
in terms of the criteria established by the accrediting agencies and by the consensus 
of eji^erts consulted for this research? 

In order to answer these two questions, the researchers used a modified 
three-phase Delphi design. Initially, a panel of 22 tcchnolo^ education leaders was 
selected and asked .o suggest criteria which could serve to indicate that the change 
to technology education had been accomplished effective^. The panel also proposed 
assessment procedures which could be used in the evaluation process. 

In the second phase, the independently-su^ested criteria and procedures 
were compiled and submitted to the panel men*«rs for their rankings of the 
importance of the items and for their editorial sug|jgestlons. From these responses, 
the researchers prepared the initial dra ft of the assessment instrument the 
Technology Teacher Education Checklist fTTEC). The re search ers compared the 
pooled consensus from the panelists with the ITEA/CTTE/NCATE guidelines 
Untemational Technology Education Association, 1987). Items selected for inclusion 
in the TTEC were (a) criteria which were highly ranked by the panel but not 
addressed by NCATE curriculum guidelines; (p) correlated to NCATE curriculimi 
guidelines for technology teacher education and distinctly different from usual 
practices in industrial ans teacher education; and (c) considered essential to support 
the process of change in an organization. 

In the third phase of the study, the draft of the TTEC was sent to the 
panelists for editorial suggestions and additional comments. The TTEC was then 
revised to reflect the suggestions and comments of the panel. 

The Technology Teacher Education Checklist, which was the primary outcome 
of this research, should be usefial to the faculty of a technology teacher education 
program or to an external evaluator in conducting formative or summative 
assessments of the change to technology education. While its use requires minimal 
duplication of the NCATE approval procedures, the items in TTEC focus upon the 
key Indicators of effective change to technology teacher education. The TTEC might 
be especially useful in a review of a technology teacher education program, a year or 
two in advance of the preparation of a curricultun folio to be submitted for 
consideration for NCATE approval. 
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Introduction 

Diirlng the summer of 1989. the Research Committee of the Council on 
Technology Teadier Education (CTIE) requested proposals for research that would 
investigate the infusion of teomology education into existing teacher education 
prograroa. The RFP asked researchers to address the problem of assessing the 
effectiveness of the change from traditional industrial arts teacher education to 
technology teacher education. In October. 1989. a small grant was awarded by 
CTTE to provide partial support for this study. 

As technology teacher education is implemented in the teacher education 
institutions in the united States, it is important to obtain an accurate assessment of 
the effectiveness of the innovation. Change in the teacher education curriculum 
may be assessed in a nimiber of possiole ways, each with several potential 
advantages. However, there is no generally accepted model for assessing the overall 
effectiveness of such a major change in technology teacher education. The goal of 
this study, therefore, was fhe development and verification of a set of measures that 
could be used to assess the effectiveness of the move to technology teacher 
education. 

Spedflcally. the study sought answers to two research questions: "What 
measurements should be used to determine the effectiveness of the change?" and 
"How should these measurements be validated?' A modified Delphi design was used 
for the study, with a panel of 22 leaders in technology education who provided input 
at all phases of the research. 

Three approaches identified through the review of literature were used in 
developing a fi^mework for a set of measurements to assess the chance to 
technology teacher education. First, the work by Hall and Horde (1987) on 
innovation configuration provided guidance in determining whether an innovation is 
actually in place. In the present case, the question became. "Has the 
implementation reached the stage where the program, taken as a whole, is 
technology teacher education, rather than inausolal arts teacher education. 
industri3^ teacher education, vocational teacher education, or technical teacher 
education?" 

Second, evaluation research by Avers, Gephart, and Clark (1989). Stufflebeam 
and Shrinkfleld (1985). and Provus (1971) framed the assessment question in terms 
of performance standards. This body of work suggested the research question. 'To 
what degree has the program moved effective^ to accomplish the change to 
technology teacher education in terms of the criteria established by the accreditii^ 
agencies and by the consensus of experts consulted for this research?' Third. 
Rogers (1983) identified concerns that aiTect innovations in organizations. FYom this 
perspective, the pertinent research question is, 'To what stage has the 
implemf^ntation process progressed?' 

The researchers sought to combine the advantages of the three approaches to 
assessment without duplicating the procedures required for accreditation. The 
criteria and procedures which were developed as a part of the study are described in 
this report. These criteria and procedures are avafiable for departmental faculties to 
use in evaluating the effectiveness of change in their individual institutional settings. 
In this application, the procedures should be particular]^ useful for self*studies one 
to three years in advance of the preparation of a folio for ITEA/CTIE/NCATE 
accreditation. In addition, the criteria and procedures could be used in comparative 
studies of the change to technolofir education across institutional boundaries. 
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Background 

Tlie literature relevant to the assessment of change and program 
implementation may be categorized into three areas: (a) educational program 
evaluation: (b) program evaluation in hi^er education, specifically in teacher 
education: and (c) change and program implementation in teacher education 
programs. Studies in eaoi of these areas were reviewed to establish the research 
base for the development of the formative evaluation system for technology teacher 
education programs. 

Educational Program Evaluation 

Ntunerous evaluation approaches may be used to father data on prograin 
effectiveness. In a literature search for an applicable model for the evaluation of 
teacher education programs. Avers. Gephart. and Clark (1989) j-cported 
"approximately 40 references to evaluation models" (p. 14). Stuflflebeam and Webster 
(1980) identified and assessed 13 alternative evaluation approaches in terms of their 
adherence to the deiinitlon: "an educational evaluation stucfy is one that is designed 
and conducted to assist some audience to Judge and Improve the worth of some 
educational object" (p. 6). Their analysis resulted in three categories of evaluation 
studies: (a) politically oriented or pseudo evaluations: (b) question oriented, or 
quasl-evaluatlons: and (c) values oriented, or true evaluations. Stufflebeam and 
Webster addressed the strengths and weaknesses Inherent in each evaluation 
approach in order to provide evaluators with a variety of fi^ameworks for conducting 
evaluation studies. 

Curriculum texts such as those by Armstrong (1989) and Glathom (1988) 
typically discuss the major educational evaluation models. Additional discussion of 
specific models is readily available (Stufflebeam and Shrinkfleld. 1985: Madaus. 
Scrlven. & Stufflebeam. 1983: Guba & Lincoln, 1981; Stufflebeam & Websten 1980: 
Popham. 1975). However, no detailed analysis of these evaluation models Is 
mcluded in this report. As Popham (1975) noted, comparing evaluation approaches 
in order to select tne best model Is usually a fruitless endeavor. Popham stated: 

Instead of engaging in a game of "sames and differents." the 
educational evsuuator should Become suffldenthr conversant with the 
available models of evaluation to decide which, if any to employ. Often, 
a more eclectic approach will be adopted whereby one selectively draws 
from the several available models those procedures or constructs that 
appear most helpful, (p. 21) 

Cronbach (1982) echoed this need for eclecticism by noting that "the 
[evaluation] design must be chosen afresh in each new undertaking, and the choices 
to be made are ahnost irmumerable" (p. 1). Indeed, an eclectic approach seemed 
most appropriate for the formative evaluation of the change to tecimology teacher 
education, given the goals of the study and the assumptions under which it was 
conducted. The review of the evaluation literature identified two approaches that 
could be comblnwi to develop appropriate instrumentation and procedures. These 
were the Context. Input Process, and Product (CIPP) Model originated by 
Stufflebeam et al. (1971). and the Discrepancy Model proposed by Provus (1971). 
These models have many commonalities. Both models: 

1. Were conceptualized and developed in the late 1960s In response to the 
need to evaluate projects funded through the Elementaiy and 
Secondary Education Act (ESEA) of 1965. 
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2. Represented efforts to broaden the view of educational evaluation to 
Indude more than an assessment of the terminal objectives. 

3. Empl^aslzed the systems view of the education by stressing the 
relationship between rontext Inputs, processes, and products. 

4. Emphasized the importance of collecting information on key 
developmental factors to aid decision-makers in assessing program 

Brogress 2\ a given point (Br^nkerhoff, Brethowen Huchyj. and 
fowakowski. 1983). 

5. Were concerned with the developmental aspects A program design and 
implementation, and recommended close colLtooraticn with program 
developers. 

6. Have been used in a variety of evaluation environments (Roth. 1978; 
Provus. 1971: and Stufilebeam. et aL. 1971). though they are not 
specifically designed for the evaluation of teacher education programs. 

The CTPP Model BJorkquist and Householder (1990) noted that "programs in 
which goals are accomplished are usually considered to be effective" (p. 69). In an 
overview and assessment of evaluation studies. Stufilebeam and Webster (1980) 
stated that the objectives-based view of program evaluation "has been the most 
pre^^ent type used in the name of educational evaluation" (p. 8). Indeed, prior to 
the ESEA. educational evaluation had focused upon "the determination of the degi'ee 
to which an instructional profflam's goals were achieved" (Popham. 1975. p. 22). 
However, a group lead by Stuf&ebeam purposed an evaluation process that focused 
upon program improvement by evaluating virtually all aspects of the educational 
program. Stufilebeam (1983) stated: 

FXindamentally. the use of the CIPP Model is intended to nromote 
growth and to help the responsible leadership and stafi^ of an 
institution systematically to obtain and use feedback so as to excel in 
meeting important needs, or at least, to do the best they can with the 
available resources. (Stufilebeam.. 1983. p. 118). 

In short, the CDPP Model placed a premium on information that can be used 
proactively to improve a program. A summaiy of the objectives and methods used 
py the four types of evaluation that comprise the CIPP Model is presexited in 
Figure 1. 
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Objective 



Method 



Ccatezt Evaluation 

To define the institutional 
context, to identify the target 
population and assess their 
needSt to identlRr opportunities 
for addressing tne needs, to 
diagnose proDlems underlying 
the needs, and to Judge 
whether proposed objectives 
are sufficiently responsive to 
the assessed needs. 

By using such methods as 
systems anafysis, survey, 
document review, hearings, 
interviews, diagnostic tests, 
and the Delphi technique. 



Input Bvaluution 

To identify and assess system 
capabilities, alternative program 
strategies, procedural desl^ 
for Implementing the strategies, 
budget, and schedules. 



By inventorying and analyzing 
available human and material 
resources, solution strategies, 
and procedural designs for 
relevance, feasibility and 
economjr; and by using such 
methods as literature search, 
visits to exemplary proflams, 
advocate teams, ana pilot trails. 



Objective 



Method 



Process Evaluation 

To identify or predict, in 
process, defects In the 
procedural design or its 
lnmlementallon, to provide 
information for the pre- 
programmed decisions, and to 
record and Judge procedtoral 
events and activities. 

By monitoring the activity's 
potential procedural barriers 
and remaining alert to 
tmantlclpated ones, by 
obtaining specified information 
for prograrrmied decisions, by 
describmg the actual process, 
and by continually Interacting 
with and observing the 
activities of project staff. 



Product Evaluation 

To collect descriptions and 
Judgments of outcomes and to 
relate them to objectives and to 
context, input, and process 
information; and to Interpret 
their worth and merit. 



By defining operationally and 
measuring outcome criteria, by 
collecting JudgmentL> of 
outcomes from stakeholders, 
and by rerformlng both 
qualitative and quantitative 
analyses. 



Figure 1. Comparison of Context, Input, Process, and Product Evaluation 

Adapted from StufHebeam (1983), CIPP Model for Program EvaluatiorL 
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Discrepancy Model Tht Discrepancy Model was developed by Provus to 
evaluate ESeA funded programs implementea In the Pittsburgh public schools. The 
modck was developed to be put In place as the new programs were designed and 
implemented. A systems approach was used to determine whether program 
performance met accepted program standards. Provus (1971) conceptualized a 
three-step process of program equation: 

1. Defining program standards. 

2. Determining whether a discrepancy exists between some aspect 
of program performance and the standards governing that aspect 
of the program. 

3. Using discrepancy Information either to change performance or to 
change program standards, (p. 183) 

According to Provus, this operational definition of program evaluation leads to four 
possible alternatives: (a) the program can be terminated* (b) the program can 
proceed unaltered, (c) the performance of the program can be altered, or (d) the 
standards governing the program can be altered (Popham, 1975). 

The Discrepancy Model has five stages: (a) design; (b) installation; (c) process; 
(d) product; and (e) program comparison. Provus (1971) noted that, "at each of 
these stages a comparison Is made between reality and some standard or standards" 
(p. 46). Ine first four stages are developmental In nature and designed to evaluate a 
single program. The fifui stage, which Provus designated as optional, provides 
Information for making comparisons with alternative programs. Each of the five 
stages Is reviewed below. 

1. Design. This stage Is similar in Intent to the first two stages of the CIPP 
Modei. The goal of the evaluator is to document the nature of the program by 
Including information fi'om a variety of constituents about: (a) goals and oblectives of 
the program; (b) the clients the program is to serve; (c) the resources available; and 
(d) instructional strategies that will promote the goals and objectives. 

2. Installation. Using the desl^ information compiled during the first stage 
as standards, the evaluator compares the level of fidelity of the Installed program 
with the design plan. The information gathered can then be used to adjust the 
Implementation procedure. 

3. Process. At this stage the evaluator asks whether the program is 
achieving Its enabling objectives, interim measures of the progress of the program 
in meeting objectives are compared to tlie design plan. Again, four options are 
available to decision-makers; (a) alter performance; (b) alter standards; (c) 
termination; or (d) proceed unaltered. 

4. Product. At this point, the focus shifts to the assessment of terailnal 
objectives. Hie post-instructional performance of learners is compared with program 
design standards to detect discrepancies. 

5. Program comparison. This optional fifth stage uses across-program 
comparisons to explore whether the program is worth the expense. A cost-benefit 
analysis is suggested for this comparison. 

Merging the eocduation models. With the commonalities of the two models 
previously stated and the thoroughness of the CEPP Model reviewed, one ml^t well 
ask why the two models should oe merged. The answer lies in the complementing 
strenglhs of the two models. CIPP. with its use of both quantitative and qualitative 
procedures and its emphasis on proactive evaluation, provides an overarching 
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-«ii.,aM«n modpl Because of its thoroughness. It Is also extremely expensive and 
S^"?SuSS?g L f^e^ and WeUer (1980) noted, values-orioited stud^es^ 
Claimed at assessing the overall merit or worth of a program ^^overty 
S^tSis ^or itlTvirtually lmS)sslble to assess the true worth of any object to. 
iS H?^r. tiie CIPP model ^des an excellent framework for approacfilng the 
multitude of possible variables in program evaluation. 

What does the discrepancy evaluation model add to this custoinlzed 
assessS app?oaS? Stream and Webster (1980) stated that QMCStloiv 
^^SS StidSattSt focus on program objectives or standards "are frequently 

ademiacv of information employed" to. 18). In particular, the discrepanqr rapie* 
?h^5Sied l^Pr^s adds Sree usS^ constructs to the evaluation process: 

1 The broadening of the evaluation procedure to Include the p(»sibiUty of 
SerS^Ttt^^tandards to conform ^ l^** SSS?f 

emphasis on standards external to the program, such as NCATE 
criteria, this approach seemed particular^ appropriate. 

2. The emphasis upon high-fldeUty implementation addressed major 
CO icems In the change process. 

3 The emphasis upon problem solving solutions to program performance 
SeratiSi ap^^ to be consisteiTt with the espoused pfiilosophy of 
technology education. 

Since technology teacher education programs are stiU largety in the 
ImpleiMnStton s^e?^ssments of their effectivencas could most Profitably focus 
S^S^JScs beS^iSTthc performances and standards that are concerned ijto 
fee inSS^ the S-owss^^ of the technology teacher education prograxns. Takra 
£tfeSe? it^iM reiSble to consider an evaluation approach that focuses on 
feput Siid ^ss evaluation components as Stuffiebeam uses the terms by 
comparing actual performance with defined standards. 

Program EvaluaUon In Teacher BducaUon 

Few studies have related specific program «>f S^?" JgPFJJ^gp*° JSg 
awsMsment of teacher education programs. GaUuzzo (i983) used the CIPP moaei 
^d^^led tSdS «amlr%io^^^cvalu^^^^^^ jdkiajtion 
Sram and Roth (1978) developed a handbook on the use of gje Ci^J^'g 

|£U-t*^-«odrK- that^^d^trovlJ^nffi^^^ 

w^lmSLs" f Sch wmponent of the tcachefeducation program. MEM tafegrates a 
SSSi?V<Srtew of the ^ogram structure with on-going process monitoring and a 
^tematic review of program outcomes. 

Perhaps the dearth of references in the Uterature to specific evaluation 
approaSw Sed in teSw education programs is the result of tiie emphasis placed 
X^S^itotton^^^ programs: Accreditation procedures reqv£c toat 
ttA^a eS^atton institutions pcffodically undertake ^tcmatic formative and 
si^tive^iSlo™ t£s reaUty into consldcraW Aycrs. Gcgj^^J^d 

cSS?lflS9)woSS^ the Accreditation Plus Model that Integrates the accreditation 
SScmL and SSSSgevSua^approaches. While focusing toe Natiwjal Coundl 
K^^AScJSt^n of Tocher ISucation (NCATC) standards and criteria for 
Sn^Sce; the model suggests a process tiiat to "active, continual, and formative 
(p. 16). The authors stated: 
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This more active evaluation stance should produce a more useful 
and higher quality evaluation. This in turn may cause the institution 
to reach out for information beyond the usual accreditation 
information. 

At this point the education unit has a decision to make. If the 
education imit is satisfied with Uie information generated via the 
accreditation process, then that docimients compliance. The design 
and planning work is done. What is left is implementation and 
monitoring. The Accreditation core of the model has been 
accomplished. 

If, however, additional evaluative questions exist and, if the 
educational unit wants those Items informed, **use-tailore(* evaluation" 
procedures will be planned and implemented. This is the ^lace for the 
Plus aspect of the Accreditation Plus Model. . . . The call here is for an 
informed eclecticism in the assembly of evaluational procedures that 
will meet the additional needs, (p. 17) 

The Accreditation Plus Model seems to be a logical extension of an already 
required practice. While this model was designed to be used for the evaluation of 
professional educational units, Uie process seems adaptable to the more specific 
evaluation concerns of technology teacher education programs. 

Change and Program Implementation 

Gee and Tyier (1976) suggested that "reasonable people will assume moderate 
risk for grtBt benefits, small risks for moderate benefits, and no risk for no benefit* 
(p. 2). while this statement makes explicit the personal nature of the change 
process, organizational characteristics are obviousfy also important factors. An 
organization, as defined by Rogers (1983), is **a stable system of individuals who 
work together to achieve common goals through a hierarchy of ranks and a division 
of labor (p. 348). The concept of a stable system imderlies the potential difficulties 
of introducing and sustaining an Innovation in an organizational environment. 

Hopkins (1984) argued that the nature of the educational organization Itself is 
a major unpediment to change. He noted that In spite of considerable external 
pressure for change in teacher education, there were few observable differences in 
the routines of professors and students. Hopkins made the provocative sxiggestion 
that '^teacher training institutions as organizations appear imable effecovety to 
manage self-initiated change" {p. 37). Giacquinta (1980), even less charitable, 
suggested that schools of education find that "change is a necrasaiy, often bitter pill 
taken for the sake of survival" (Hopkins, 1984, p. 43). These opinions seem to be 
shared by several state le^slatures which have recently mandated changes in 
teacher education requirements and practices. 

In a stucty of Canadian teacher training institutions, Hopkins (1982) Identified 
a lengthy list of barriers to change. The barriers were categorized as dthen (a) 
systemic: (b) organizational: or (c) individual. Also, barriers were classified either as 
real or perceived. In distinguishing between real and perceived barriers, Hopkins 
stated, iis the research processed, it became obvious that a number of the barriers 
to change were perceived rather than actual, or . . . were a reality that had little 
tmp^cSi support" (p. 20). Hopkins* listing of types of barriers is presented in 
Figure 2. 
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Systemic 



Organlsatloaal 



Individual 



Organlutlonal Change 

- Economic factors 
. Political pressure 

teovemment) 

- VulnerabiUty 
(environmental constraints) 

- Central university 
administration 

- Tradition 

- Lack of dear mission (goal 
variabiUty) 

- Incongruent reward system 

- Poor communication 

- Absence of linking 
structures (low 
interdependence) 

- Autonomous pluralism 
(professorial Independence) 

- Complexity of decision- 
maUng process 

- Inadequate implementation 

- Discontinuity of personnel 
(academic ycar» 
sabbaticals* retirement) 

- Inertia (acceptance of the 
status quo) 

- Poor leadership 

- Tenure system 

- Pluralism of roles 

- Socialization into a 
discipline (identification 
with discipline as opposed 
to institution) 

- Incompetence 

- Innovation fatigue 



Perceived Barriers 

- Inertia 

- Future imcertainty 



Bureaucratic myth 



Top down approach to 
change 



- Teachers* college ItgjBicy 



- Recent history of change 



Emotional resistance to 
change 

Too busy (high status of 
busyness ethos) 
Tendency to extemali/e 
(blame others for the state 
of inertia) 



Figure 2. Barriers to Change In Teacher Education 

is (1982) » The Probl 
Teacher Education. 



Adapted from D. Hopkins (1982)»_7he Problem of Change in 
Canaxuan^ 
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Understanding the barriers to change is critical to the success of an 
innovation. An accurate anaJvsis of the barriers to change provides a staiUng point 
for attacking the problem. Ad Sarason (1971) noted^ "recogplzlng the adversary gives 
sint a basis for assexllng that the problem neither hopeless nor insoluble^ (p. 
236). 



A Mv^el for Organixational Change 

In a review of the literature on innovation in higher education. Dill and 
Friedman (1979) observed that "it is voluminous . . . land) overwhelmingly 
descriptive" (p. 412). The authors described four frameworks that addressed change 
in hlgner education: (a) complex organization model; (b) conflict model: (c) difiusion 
modd: and (d) planned change model. Dill and Ftledman speculated that a more 
powerful eclectic model may emerge from a dear tmderstandlng of each of the 
frameworks analyzed. Moreover, the authors suggested that the cxplanatoiy power 
of an amalgamated model would be enhanced if the strengths of the difhision model 
were incorporated into the research desijBi and lay taking time and the 
cxtensiveness of adoption into explicit account*^ (p. 433). 

A model of the innovation-decision process in an organization* developed by 
Rogers (1983), addresses these factors from another perspective. The Rogers model 
was synthesized from the dlfTusion research tradition and focuses on the process of 
adoption, implementation, and the incorporation of the innovation into the 
organization. The five steps in the model are divided Into two stages: initiation and 
implementation. 

Initiation Stage. During the Initiation stage, organizational activities center 
aroxmd the informatlon-gathertag, conceptualizing, and planning that is reauired to 
make the decision to change. Tne two steps Included at this stage are: (a) agenda 
setting, where the Initial Idea search occurs and the motivation to change is 
generated; and (b) matching, where organizational problems and possible solutions 
are analyzed for compatibility. 

The Initiation stage is essentially a problem solving exercise. As the 
organization becomes cognizant of a performance shortfall, it initiates a search of the 
environment for possible solutions to the problem. For exranple, industrial arts 
programs were generally faced with declining enrollments. At the same time, many 
studies dted the need for students to possess increased scientific and technological 
literacy. In response, the field started to focus on technology education as an 
emergent solution to both problems. 

Iniplementation Stage. The second stage. Implementation, begins after the 
decision to make the change has been made by the organization. This stage 
includes the decisions, actions, and procedures Involved In putting an Innovation 
into regular use. The implementation stage includes three steps: (a) 
redefining/restructuring the innovation and the organization to accommodate the 
change: (b) clarifying tile iiinovatlon as it is put into regular use: and ultimately (c) 
routlnlzlng or Institutionalizing the change as an integral part of the ongoing 
activities of the organization. 

According to Rogers (1983) each step is • characterized by a particular range of 
events, actions, and decisions" (p. 362). Further, the latter steps carmot occur imtil 
the Issues in the earlier steps have been resolved. Citing the work of Pelz (1982) as 
a source of support for the model, Rogers noted that Iimovations Iniported into an 
organization -usually occur in the time-order sequence" (p. 366). However, 
iimovations that originated within an organization are not characterized by a 
similar^ clear pattern of adoption. Since technology teacher education programs are 
currently changing in an attempt to meet lar^y external Irmovatlons (NCATB 
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accreditation standards and state certification requirements). It appears that iat 
time-order sequence Is expected to appty. Hie Unear nature of the Innovation- 
decision model hlghll^ts the need to nurture the change to technology teacher 
education throu^out me stages of the entire change process. 

Suimnarj 

In light of the review of literature and the spedflc goals of this research effort 
the decision was made to develop an evaluation design incorporating an eclectic mix 
of program evaluation approaches, the NCA1E accreditation process, and 
descriptions of the process of change as that process may be expected to occur In 
teacher education organizations. Stufflebeam's CIPP Model provided an overall 
framework from whichto assess the effectiveness of change to technology teacher 
education. Provus's Discrepancy Model added the possibility of acyustlng the 
measurement standards to conform to program performance reality. And, because 
accreditation is an overarching evaluation concern for teacher education, the 
Accreditation Plus Model suggested a way of Integrating program evaluation and 
accreditation. Further, because technology teacher education programs are prwently 
In the early Implementation stage, measures that reflect the process of change 
seemed to be appropriate for Inclusion. 



Methodology 

The goal of this study was the development and validation of a set of 
measures that could be used by teacher education faculty and/or by external 
evaluators to assess the effectiveness of the change from an Industrial arts teacher 
education program to a technology teacher education program during the 
Implementation of that change. Although a full scale formative evaluation would be 
required for the most accurate assessment, several delimitations appty to the 
formative evaluation design developed by this project 

1. The time required for on-site data collection by the external evaluator(s) 
should not exceed two observer-days. 

2. Wth the exception of Interviews and classroom and laboratory observation 
sessions, the data ^thering should not require additional faculty time. 

3. Existing data should be used whenever possible. 

4. Data gathering should not seriously disrupt on-going Insbnctional 
activities. 

In order to design a formative evaluation system which could be employed 
within these delimitations, three assumptions were necessary. The first assumption 
accepted the vali^iihr of the phlloso^y. mission, and goals of technology teacher 
education as stated "by International Technology Education Association (TIEA) (1985) 
and delineated In the criteria of the National Council for Accreditation of Teadier 
Education (NCATE) (1987). Second, it was assumed ttiat the technology teadier 
education program has applied for or Is working toward NCATE approval. Third, It 
was assumed that the rationale and priorities of the technology teacher education 
program are congruent with the needs of its constituencies. 

V^thin these assumptions and delimitations, two questions were asked. First, 
as existing teacher education programs are modified to prepare technology education 
teachers, what measurements should be used to determine the effectiveness of the 
change? Second, how should these measurements be validated? 
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Procedures 

To answer the first question, the researchers used a modified Delphi desifli. 
Nomimirions of leading practitioners and advocates In technolog y edu cation who 
might nerve as Delphi panelists were solicited from officers oT CITE and ITEA 
(Appendix A). Ihis process resulted in the selection of a pan el com prised of the 22 
ttulMduals who were recommended by at least two of the CTTB or ITEA officers 
(Appendix B). 

An open-ended questionnaire was sent to the panel members. Panelists were 
asked to suggest criteria for evaluating the elBectiveness of the change from 
industrial arts teacher education to technology teacher education programs. Space 
was provided on the questionnaire to enter 10 crtteria. but the panelists were 
encouraged to list as many criteria as they felt were necessary (Appendix C). 

Panelists were also asked to suggest procedures for evaluating the 
effectiveness of the change from industriafarts teacher education to technology 
teacher education programs. Five response blanks were provided, but panelists were 
encouraged to Include as many evaluation procedures as they thought appropriate. 

Fourteen panelists returned the first round questionnaire. Initial responses 
were tabulated, duplications were eliminated, and similar suggestions were 
combined. These procedures resulted in a list of 58 criteria and 33 procedures for 
evaluating the effectiveness of the change to technology teacher education. The 
criteria were sorted into four categories: (a) the tecnnolpcy teacher education 
program, (b) faculty members, (c) student skills, and (d) capablilUes of graduates. 

The second round questionnaire asked the 22 panelists to rate the importance 
of the 58 criteria and 33 procedures on a scale which ranged firom 0 to 10. The 
instructions defined a rating of 0 as a reconmaendation that the criter jn or 
procedure be dropped. A rating of 10 meant that the criterion or procedure was 
considered to be absolutely vital lo the assessment of the effectiveness of the change 
to technology teacher education. Panelists were asked to offer editorial su^estions 
on the statements of criteria and procedtires and also to suggest addltionaTcriteria 
and procedures (and to rate any additional statements) (Appendix D). 

Eighteen of the 22 questionnaires were returned promptly. The 18 responses 
were tabulated and the mean rating of importance for each item was calculated. 
The statements of criteria and procedures were then listed in order of their mean 
rating of Importance. The raiiked listings for each critertcm with a mean value 
greater than 9.0 on the 10 point scale are Included in Table 1. The complete 
ranked listings are reproduced in Appendix E. 



Table 1 

listing qfMghly Ranked Criteria and Procedure Statements Sorted by 
Quesaonncdre Category. 



Mean Technology teacher education program 

9.55 Laboratory Instruction provides opportunities for students to reinforce 
abstract concepts with concrete experiences. 

9.50 Instructional strategies emphasize conceptual understanding and problem 
solving. 
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Table 1 Continued 

9.23 Professional studies coirawnent emphasizes the study of technology, 
including 30Clal*cultural anects and consequences. 

9.22 Laboratories facilitate the learning of broad based technological concepts. 

9.22 Instruction incorporates current technological activities. 

9.17 PhUosophVt mission statement, goals and curriculum emphasize 
technolo^cal sidlls as opposed to technical skills. 

9. 17 Social-cultural Impacts of technology are emphasized. 

9. 12 Field experiences are technology centered. 

9.05 Problem solving and decision making abilities are emphasized. 

9.00 Curricula are based on recent research findings. 

Fuculty Members 

9.50 Display a positive attitude toward the technology teacher education 
curriculum. 

9.22 Participate in plarmed professional development activities to update their 
knowledge and skills. 

9.05 Communicate their tinderstandlng of the meaning and impllcaUons of 
technology education both within and outside the classroom. 

Students are expected to: 

9.78 Be people oriented. 
9.44 Be future oriented. 

9.39 Demonstrate the ability to teach problem solving techniques. 

9.33 Effectively plan and implement technology education in grades 5-12. 

9.28 Develop and implement curriculum material that reflect a broad 
technological system area. 

9.28 Demonstrate an awareness of societies reliance on technological s)rstems, 

9.22 Plan and Implement teaching-learning activities. 

9.17 Use a vocabulary that reflects the concepts of technology education. 

9. 1 1 Apply current instructional theory. 

9.06 Formulate appropriate objectives. 

9.05 Be open to change and willing to Initiate change. 
9.05 Consider global perspectives in technology education. 
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Table 1 Continued 

9.00 Demonstrate a basic v iderstanding of tools, machines and process and 
their applications in manufacturing, construction, communication, and 
transportation. 



Graduates of the technologjr teacher education program: 

9.78 Employ a philosophy which reflects a technological base. 

9.61 Teach concepts and use teaching techniques that are technology based. 



Procedure Statements 

9.50 Examine the currlculvmi to determine if the philosophy, definition, mission 
statement, goals and objectives, course cont'-nt. and learning experleiice 
reflect technology education. 

9.22 Analyze the courses required in the program, the content contained in each 
of the courses, teadilng strate^es ana methods, assignments, tests, and 
student field experience to determine if they reflect technology education. 



Several weeks after the data analysis was completed, three additional 
completed questionnaires were received. Inspection of these questionnaires Indicated 
that the ratings of item importance were essentialty comparable to the results of the 
data anatysls. Since the preparation of the Instrument was nearly completed at that 
time, the data from the three late respondents were not Included in the criteria and 
procedure rankings. 

Selecting the Measurement Criteria and Procedures 

One of the major outcomes of this research effort was the idenllflcsitlon of a 
high degree of correspondence • between the criteria for assessment of teclmology 
teacher education programs suggested by the panel of eiqperts in this studv and tTie 
NCATE approved curriculum ftiidelines as specified in Basic Program in Terhnolog j 
Education (1987). Most of the remaining criteria suggested by tlie pard 
corresponded closely to the guidelines firom the Standards, Procedures, and Poiides 
for the Accreditation of PtXiJessional Education Units (NCATE. 1987). Hie criteria 
sxiggested by the panel of ejqserts in this study and the NCATE curriculum 
guldellne(s) most closely associated with the respective suggested criteria are listed 
in Appendix F. 

Developing the Technology Teacher Education Checklist (TTEC) 

An initial review of the listing of criteria and procedures identified by the 
panelists in this research suggested many parallels to the NCATE accreditation 
process. The primary Intent of this Investigation was not to duplicate the NCATE 
assessment process, but to identify essential elements in the Implementation of 
technology teacher education that would seive as key Indicators of me efiectlveness 
of the change from industrial arts teacher education. In order to concentrate the 
assessment eflbrt, therefore, items were selected for Inclusion in the measurement 
Instrument If they were: 

1. HlCWy ranked within their criteria category but not addressed by 
NCATO curriculum guidelines; 
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2. Correlated to NCATE currlculiun guidelines for technology teacher 
education and distinct^ different from usual practices in industrial arts 
teacher education; or 

3. Considered to be essential to support the process of organizational 
change. 

The first draft of the TIEC did not include measurements of program 
outcome, such as performance of program graduates. Ihese items were excluded 
from the measurement instrument since techndogy teacher education Is in the 
implementatton phase, a stage when Hall and Herd (1987) noted that Interpreting 
any outcome data is extremely rlsly" (p. 343). The draft Technology Teacher 
Education Cheddist is reproduced in Appendix G. 

Verification of the TTEC 

In order to verify the measures selected for inclusion in the checklist a draft 
of the TIEC was sent to the panel for editorial suggestions and additional 
comments. Sixteen of the twenfy-two panelists responded. Most respondents 
suggested editorial revisions or made other comments. Caref ul consideration was 
given to these suggestions as revisions were made in the TTEC. One of the major 
changes as a result of the verification process was the inclusion of several program 
outcome measures. Panelists strongly recommended that a numb er of outcome 
measures which were initially highly-ranked should be included in the TTEC. despite 
the difiacultles in interpreting oufout data. Similarly, involvement in a TECA chapter 
was included in the revisions of the TIEC in response to suggestions from panelists. 
The TIEC. revisv*d to incorporate suggestions torn panelists, is reproducca in Table 
2 below. Suggestions and the comments received from the panel about the draft 
TTEC are included in Appendix H. 



Table 2 

Technology Teacher Education Checklist. 



A. Examine the catalog, a sample of curriculum documents, and a sample of 
course syllabi to vertf^ the degree to wh^ch: 

1. The philosophy, mission statement, and goals and objectives o(tiic 
program reflect tl:e definition(s) of technology education suggested by ITEA. 
CTlte. and relevant groups in the state/province. 

2. Study is required in technological systems such as communication. 

E reduction (construction and manufacturing), transportation, and 
iotechnology. 

3. Courses in mathematics, science, and computing science are required. 

4. Required full-time student teaching and early field experiences are 
conducted in an exemplary technology education setting. 

5. Required reading lists provide compreliensive coverage of technology and 
technology education. 

6. Learning activities and experiences are representative of technology 
education. 
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Table 2 Continued 

B. Interview the department head with regard to the change to technology teacher 
education to discern the degree to whioi: 

1. Funding is adequate to support the current technology teacher education 
program and plans are in place for periodic replacement and upgrading oV 
tkdJities and equipment. 

2. Faculty and staff allocations are adequate to serve student enrollments in 
technology teacher education. 

3. The written departmental plan for faculty professional development and 
technolo^cal updating is adequate to prepare Haculty members for 
contemporary technology teacher education. 

4. Enrollments in the n^jor are adeqtiate, stable, or increasing. 

5. The written departmental implementation plan for technology teacher 
education addresses the process of organizational change. 

6. Faculty are committed to the philosophy and objectives of technology 
educauon. 

C. Interview faculty members and review recent aimual reports, blodata 
information, faculty publications, copies of presentations, and manuscripts being 
considered for publication to verify whether 

L Faculty are writing scholarly papers, developing instructional materials, 
and giving presentations about tecimology education. 

2. Current faculty research and service activities are directed toward topics 
and Issues in technology education. 

3. Faciilty are actively involved in professional organizations in technology 
educauon. 



D. Observe professional and technical classes to discern the degree to which: 

1. Instructional methods emphasize technological problem solving and 
decision-making. 

2. Instructional materials reflect contemporary technology. 

3. Major elements of technology education (e.g.. sjrstems. environmental and 
sodal impacts, and the applications of technological devices) are 
emphasized in the course activities. 



E. Inspect laboratory facilities to ascertain the degree to which: 

1. Laboratories are adequate for effective instruction. 

2. Equipment and space provide students adequate opportunities for 
experiences in state-of-the-art applications of technology (e.g.. CAD/CAM. 
CIM. robotics, desk-top publishing, lasers, table-top technology, 
hydroponics). 
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Table 2 Continued 

F. Interview students, and examine student logs and required student work to 
discern whether 

L The elements of technology education are understood and Integrated into 
their total philosophy of education. 

2. They are active in a TECA chapter. 

3. The problem solving process and decision-making rationale are 
Incorporated into grading. 

4. Environmental consequences and social-cultural effects of technology are 
reflected in student activities. 



G. Interview chairs of related departments and administrators (dean, provost or 
president) to ascertain the degree of philosophical support that is provided for 
technolo^ education. 



H. Listen to conversations and discussions and obser/e student activity to discern 
the de^ee to which: 

L The terminology used by faculty and students reflects technology and 
technology education. 

2. Faculty and students appear to be enthusiastic about technology 
educauon. 



I. Interview principals who have experience with student teachers and graduates of 
the technology education program to discern whether the program prepares 
professionals to: 

1. Plan and implement technology education. 

2. Use problem solving strategies. 

3. Apply current instructional theory. 



Using the Instrument 

Jordan (1989) began a discussion of evaluation and change by reminding 
practitioners that 

One of the axioms of measurement Is that assessment Is not an end in 
itself. We evaluate because we wl^ to know the current state of affisdrs, 
but we wish to do that in order to make improvements. Exactly how we 
wish to improve depends on what we discover. In theory, the process is 
circular and imending. That is, we should assess and m£kt improvements 
and then assess the improvements, (p. 147). 

With this interaction between evaluation and change in mind, there are several 
possible ways of using the instrument developed through this research. Perhaps the 
simplest use would be for an internal or external evaluator to use the Instrument as 
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a checklist of what has been accomplished and what Is In progress (or still to be 
initiated). Two more complex uses may Include determining if the innovation is in 
place and using force field ana^^ to determine sources of resistance. 

Determining if the Innovation is In-Place 

Hord, Rutherford. HuUng-Austln and Hall (1987) proposed that bcrorc asoesslng 
program outcomes it is first necessary to determine that the innovation is in fact in 
place. They indicated two ways of making that determinatioxu (a) first the level of 
fidelity of fne actual implementation of the Innovation can be compared with the 
intenaed Innovation by determining the innovation configuration, and (b) second, the 
actual levels of use can be determined. According to the authors: 

Innovation Confijguratlon (IC) represents the patterns of Innovation use that 
result when different teachera put innovations into operation in their 
classrooms. In the course of our early work, we noted that IndMdual 
teachers (and professors) used dlfierent parts of an ixmovation in dSiierent 
ways. When these parts were put together, a number of patterns emerged, 
each characterizing a dlfierent use of the Linovation. We called these 
patterns Innovation Configurations, (p. 13) 

Herd et al. (1987) proposed that each innovation has essential and related 
components. The essential components cannot be changed without, undermining the 
natiure of the Innovation itself. The related concepts aUow for local flexibUity and, 
whfie varied, are still faithful to the Innovation deslffli. Hord et al. suggested that 
assessment of fidelity can be made by developing an iC checklist that outlines ideal, 
acceptable, and unacceptable variations of the innovation. In technology teacher 
education programs, many of the criteria identified through this research may serve 
as the "essen&l" components. 

The second measure proposed by Hord et al. (1987) to determine whether or not 
the Innovation Is actually in place is an assessment of the six levels of use. These 
levels range fi-om Level of Use 0. nonuse, to Level of Use VI, renewal, where the 
"user reevaluates the quality of use of the Innovation, seeks major modifications of 
or alternatives to, present Innovation to achieve Increased Impact on clients, 
examines new developments in the field, and explores new goals for self and the 
organization" (p. 55). By using the TTEC to identify the essential components of the 
change to technology teacher education, an assessment of levels of use fi'om the 
perapective of the feculty may be an Important step in measuring the effiectlveness of 
the change and planning lurOier intervention strategies. 

Force Fiela Analysis 

Lewln (1951). the originator of field psychology, proposed that change Is the 
result of compeUtlon between driving and resisting forces (Daft. 1988). Lewln's 
conceptualization has been adapted to describe the dynamics of a number of 
management situations In organizational change. Daft stated that: 

To Implement a change, management should analyze the chaise forces. By 
selectivefy removing forces that restrain change the driving forces will be 
strong enough to enable implementation. ... As restraining forces are 
reduced or removed, behavior will shift to Incorporate the desired changes, 
(p. 313) 

Mfiler (1987) suggested that force fidd anafysls could be used to nurture a climate 
receptive to innovation and creativity. Miller stated: 

The prlmaiy fimction of the force field In idea gen^^ratlon Is to present tlj-ee 
dlfierent stnnull for thinking of new options or solutions. Because the field 
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represents a kind of tug-of-war. there are three ways to move the center line 
in the direction of the more desirable future: 

1. Strengthen an already present positive force. 

2. Weaken an already present negative force. 

3. Add a new positive force, (p. 73) 

If these two ideas are taken together, a picture emerges of how force field 
analysis and the instrument designed through this research could be applied to the 
transition from industrial arts teacher education to technology teacher education. 
First, each criterion could be assessed to determine its relative strength as a drtvlng 
force for change. Second, forces unique to the particular implementatioa may be 
identified and dealt with. Third, the Information generated through the assessment 
could be used to strengthen the Implementation procedures. In this way. the 
Instrvmient may serve as a game plan for implementation and continued assessment 
of the change. 

Next Steps 

The Technology Teacher Education Checklist the prlmaiy product of this stuchr. 
n / appears to be rea(ty for field testing and validation. The researchers plan to 
conduct field testing during the 1990-91 academic vcar by applying the TIEC 
criteria and procedures in a sample of technology teacher education programs. 
Results of the TTEC field testing will be validated by comparisons with other 
measures of program quality. 
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Letter to ITSA end CTTS Officers Soliciting 
Nomlnitlons to the Penel 



November 22, 1989 



Dear 

The Council on Technology Teacher Education has recently awarded me a small 
fflant to assess the effect^eness of the change to technology teacher education as 
Oiat change is implemented. The research drngn and procedures developed imder 
the grant will be available for departmental acuities tc use In evaluating the 
efifecSveness of change In their own institutional settings. In addition, the research 
design and procedures should be useful in comparative studies of the eflfectlveness 
of the change to technology teacher education across institutional boundaries. 

A modified Delphi desl0i is being used in the development of the research 
instruments. A panel of leading practitioners and advocates in technology teacher 
education will be asked to suggest ways in which techndo^ teacher education 
programs should be assessed. Nominatio ns o f individuals lo serve as Delphi 
panelists are being sou^t firom officers of CTTE and ITEA, in accordance with the 
procedures outlined in the proposal. 

Please list on the enclosed form at least 10 individuals who could assist the project 
in identi^g criteria and procedures for assessing the effectiveness of the change to 
technology teacher education programs? A return envelope Is enclosed. It is perfectly 
appropriate to list yourself as one of the possible resource persons. 

Thanks for taking time to assist in this first phase of the research. 

Sincerely, 



Daniel L Householder 
Professor and Project Director 
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Listing of PinelltU 



Dr. LowcU TX Anderson 
Department of Industrial Technology 

Education 
School of Technology 
Indiana State University 
Ttnt Haute, IN 47809 

Dr. Myron Bender 

Department of Industrial Technology 
Couege of Human Resovirce 

Development 
University of North Dakota 
Grand Forks, ND 58202 

Dr. Rogf r U. Betts 
Deparmie^A of Industrial Technology 
CoUege of Natural Science 
University of Northern Iowa 
Cedar Falls, lA 50614-0178 

Dr. William E. Dugger 
Technology Education Program Area 
Vocational-Technical Education 
Virginia Potytechnic Institute and State 

ffiiiverslty 
Blacksburg, VA 24061-0432 

Dr. Thomas L. Erekson 
Industrial, Technological and 

Occupational Education 
College of Education 
University of Maryland 
CoUege Park, MD 20742 

Dr. Everett N. Israel 
Department of Industrial Technology 
CoUege of Technology 
Eastem Michigan university 
YpsUanti, MI 48197 

Dr. Donald P. Lauda 
AppUed Arts and Sciences 
Califomla State University 
Long Beach, CA 90840 

Dr. Franzie L. Loepp 
Illinois State University 
Department of Industrial Technology 
CoUege of AppUed Science and 

Technologyr 
Normal, IL 61761 



Dr. G. Eugene Martin 
School of implied Arts & Technology 
Southwest Texas State University 
San Marcos, TX 78663 

Dr. David L. McCroiy 
Technology Education Program 
C(dlege oTHuman Resources and 

Educaticm 
West Virginia University 
Moigantown, WV 26506-6122 

Dr. Douglas L. Polette 
Technology Education, Cheever HaU 
CoUege of Agriculture 
Montana State University 
Bozeman, MT 59717 

Dr. John M. Ritz 

Occupational and Technical Studies 

Department 
CoU^e of Education 
Old uominlon University 
NorfoUc, VA 23529 

Dr. James H. Rokusek 

Department of Business and Industrial 

Education 
CoUege of Technology 
Eastem Mlchi^ lAiiverslty 
YpsUanti, MI 48197 

Dr. Ernest N. Savage 
CoUege of Technology 
Bowling Green State University 
BowUng Green, OH 43403 

Dr. Anthony E. SchwaUer 
Department of Industrial Studies 
Headley HaU 

CoUege of Science and Technology 
St Cloud State University 
St. Cloud, MN 56301 

Dr. Donald F. Smith 
Department of Industry and 

Technology 
CoUege of /^pUed Sciences and 

Teomolofir 
BaU State University 
Munde, IN 47306 



Dr. Kendall N. Starkc^'cather 
International Technology Education 

Association 
1914 Association Drive 
Reston. VA 22091 

Dr. Leonard F. Steny 
Scho(d of Industry and Technology 
Unlversily of \Knsconsin-Stout 
1102 Union Street 
Menomonie. WI 54751 

Dr. Ronald Tuttle 

School of Business and Technology 
Kearney State College 
Kearney. NE 68849 

Dr. A. Emerson Wlens 

Department of Industrial Technology 

CoUege of Applied Science and 

Teomology 
Illinois State University 
Normal. IL 61761 

Dr. John R Wright 

School of Technology 

Central Connecticut State University 

1615 Stanley Street 

New Britain. CT O6050 

Dr. R. Thomas Wright 
Department of Industiy and 

Technology 
CoUege of Applied Sciences and 

Teomology 
BaU State University 
Munde. IN 47306 
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Appendix C 

Round I Letter and Open Bnded Ouettionnalre to Penelltte 



December 18, 1389 



Dear Panelist: 

Itie Council on Technology Teacher Education has recently awarded roe a amall 
tfrant to assess the effecttveness of the change to technology teacher education as 
uiat change is implemented. The research design and procedures developed under 
the grant will be available for departmental fisiculties to use in evahiating Che 
efferaveness of change in their own institutional settings. In addition, the research 
design and procedures should be usefiil in comparative studies of the effectiveness 
of the change to technology teacher education across institutional boundaries. 

A modified Delphi desioi is being used in the development of the research 
instruments. You have oeen nominated to serve on a panel of 22 leading 
practitioners and advocates in technology teacher education who are being asked to 
suggest ways in which the effectiveness of the change from industrial arts teacher 
education to technology teacher education programs should be assessed. 

Please list on the enclosed form at least five criteria and at least three procedures 
for assessing the effecUveness of the change to technology teacher education 
programs. A return envelope is enclosed to fecllitate your response. 

Thanks for your participation In this phase of the research. As soon as the results 
of this round are compiled, you will be asked to prioritize the criteria and 
procedures as the new step in the development of the evaluation process. Your 
CTTE colleagues - and their students will benefit from your contributions to this 
eflbrt. 

Sincerely. 



Daniel L. Householder 
Professor and Project Director 
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Round I Open Ended QuMtionnaire 

Criteria for evaluating the effectiveness of the change from industrial arts teacher 
education to technology teacher education programs: 

1. 
2. 
3. 
4. 
5. 



7. 
8. 
9. 
10. 



Procedures for evaluating the effectiveness of the change from Industrial arts teacher 
education to technology teacher education programs: 

1. 



3. 
4. 

5. 



2U 
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Appendix D 

Round n Letter and Qoestlonttalre to PaneUt ts 



January 31. 1990 



Dear Panelist: 

Thanks for your assistance as a member of the panel In our Coimcil on 
Technology Teacher Education project Hrtie Development and PUot Testing o f a 
ReseanftDeslgn for Evaluating the Effectiveness of Change to NCATE/CTTE/rTCA 
Curriculum Guidelines In Technology Teacher Education. 

Dtirlng the first round of the Delphi stuchft panelists suggested criteria and 
procedures for assessing the effectiveness of program change from Industrial arts 
teacher education to technology teacher education. These have now been tabulated, 
reviewed, and edited to create the enclosed lists. Hie next task In the research is to 
Identify the most Important criteria and the most promising procedures. 

Please review the list of criteria and edit each of them as you see fit Evaluate the 
importance of each criterion by circling a ntmiber on the 0-10 scale. Then, repeat 
this process with the list of procedures. A return envelope Is enclosed for your 
response. 

Thanks again for your continued participation In the research. Your CTTE 
colleagues - and their students - will benefit from your contributions to this efiort. 

Sincerely. 



Daniel L. Householder 
Professor and Project Director 
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Round n Qucttloiuulre 



CRITERIA 



Technology teacher education (TTE) Is the (usually) undergraduate program 
that prepares spedallzed teachers who are capable of <uttanlzlng and impTementlng 
contemporaiy programs of technology educatton flE) In the elementaiy and 
secondeuy schools. 

In ordo- to evaluate the effectiveness of the change fir<nn Industrial arts teacher 
education to technology teacher education several criteria must be enq)loyed. The 
criteria listed below were identified during the first round of conespondencc with the 
panel members. 

Hease rate the importance of each criterion by circling a nimiber on the 0 to 10 
scale. (You may edit the criterion if you wish.) A rating of 0 is a recommendation 
that the criterion be dropped. A rating of 10 Implies that the criterion is absolute^ 
vital to the assessment ot the effectiveness of the change to technology teacher 



If a criterion should be added, please write it In the space provided on page 5. 
indicate Uie categoiy where it should be added, and rate it. 



Tecbnology Teacher Education Program 

0..1..2..3..4..5..6..7..8..9..10 Courses cover the snectrum of technology education. 



0..1..2..3..4..5..6..7..8..9..10 General education requirements in mathematics. 

science, computer science, and humanities provide the 
necessary breadth and depth. 



0..1..2..3..4..5..6..7..8..9..10 Problem solving and decision making abilities are 

empha^ed. 



0..1..2..3..4..5..6..7..8..9..10 Laboratoiy instruction provides opportunities for 

students to reinforce abstract concepts with concrete 
exptriences. 



0..1..2..3..4..5..6..7..8..9..10 Instruction emphasizes the safe and efficient use of 

tools, machines, and equipment 



0..1..2..3..4..5..6..7..8..9..10 Teacher preparation is the primary programmatic focus. 



0..1..2..3..4..5..6..7..8..9..10 Curriculimi is In compliance with ITEA/CTTE/NCATE 

guidelines. 



0..1..2..3..4..5..6..7..8..9..10 Philosophy, mission statement, goals and curriculum 

emphasize technological skills as opposed to technical 



education. 
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Program Criteria • Continued 

0..1..2..3..4..5..6..7..8..9..10 Cutting edge technologies arc included in instructional 

offering. 

0..1..2..3..4..5..6..7..8..9..10 Professional studies component emphasizes the study of 

technology, including social-cultural affects and 
consequences. 

0..1..2..3..4..5..6..7..8..9..10 Library acquisitions reflect contemporary technology. 



0..1..2..3..4..5..6..7..8..9..10 Textbooks and Instructional media reflect contemporary 

technology. 



0..1..2..3..4..5..6..7..8..9..10 Field experiences are technology-centered. 



0..L.2..3..4..5..6..7..8..9..10 Instruction Incorporates current technological activities. 



0..1..2..3..4..5..6..7..8..9..10 Content Is organized around technological sjrstems. 



0..1..2..3..4..5..6..7..8..9..10 Technology, as a body of knowledge, determines 

program goals and structure. 



0..1..2..3..4..5..6..7..8..9..10 Social-cultural Impacts of technology are emphasized. 



0..1..2..3..4..5..6..7..8.,9..10 Instructional strategies emphasize conceptual 

understanding and problem sohdng. 



0..L.2..3..4..5..6..7..8..9..10 Curricula are based on recent research findings. 



0..L.2..3..4..5..6..7..8..9..10 Facilities represent contemporary technologies. 



0..1..2..3..4..5..6..7..8..9..10 Laboratories facilitate the learning of broad based 

technological concepts. 

0..1..2..3..4..5..6..7..8..9..10 Administrators in the institution afire 3 with and support 

the philosophical change to technology teacher 
education. 



0..1..2..3..4..5..6..7..8..9..10 Financial support for technology teacher education is 

adequate. 
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Protfrtm Criteria • Continued 

0..1..2..3..4..5..6..7..8..9..10 Program is recognized as tedmology teacher education 

fay fitctitty within the Institution. ooUeagues In other 
Institutions, and public school admlnlsnraton. 

Facultj Membert 

0..1..2..3..4..5..6..7..8..9..10 Participate In planned professional development 

activities to update their knowledge and skills. 

0..1..2..3..4..5..6..7..8..9..10 Are active^ engaged in research in technology 

education. 

0..1..2..3..4..5..6..7..8..9..10 Publish in the field of technology education. 

0..1..2..3..4..5..6..7..8..9..10 Present at national, regional, and local conferences. 

0..1..2..3..4..5..6..7..8..9..10 Display a positive attitude tou-ard the technology 

teacher education curriculum. 

0..1..2..3..4..5..6..7..8..9..10 Communicate their understanding of the meanmg and 

Implications of technology education both within and 
outside Uie classroom. 



Students are expected to: 

0..1..2..3..4..5..6..7..8..9..10 Identify the concepts, principles, and systems of 

technology. 

0..1..2..3..4..5..6..7..8..9..10 EflFectively plan and Implement technology education in 

grades 5-12. 

0..1..2..3..4..5..6..7..8..9..10 Apply current Instructional theory. 
0..1..2..3..4..5..6..7..8..9..10 Demonstrate knowledge of current technology. 
0..1..2..3..4..5..6..7..8..9..10 Formulate appropriate objectives. 
0..1..2..3..4..5..6..7..8..9..10 Develop or use a new or existing taxonomy. 
0..1..2..3..4..5..6..7..8..9..10 Plan and Implement teachlng-leamtog activities. 
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Student Criteria - Continued 

0..L.2..3..4..5-.6..7..8..9..10 Design evaluation devices and Instalments. 

0..L.2..3..4..5..6..7..8..9..10 Demonstrate a basic understanding of tools, machines 

and process and their applications in manu&cturlng, 
consmiction, communication, and transportation. 



0..L.2..3..4..5..6..7..8..9..10 Develop and implement currlculimi material that reflect 

a broad technological system area. 



0..1..2..3..4..5..6..7..8..9..10 Develop a curriculum that illustrates how new 

technology is created. 



0..1..2..3..4..5..6..7..8..9.,10 Develop a curriculum that analyzes the social-cultural 

affects of technology. 



0..1..2..3..4..5..6..7..8..9..10 Demonstrate the ability to teach problem solving 

techniques. 



0..1..2..3..4..5..6..7..8..9..10 Demonstrate an awareness of society*s reliance on 

technological systems. 



0..1..2..3..4..5..6..7..8..9..10 Develop conceptual understandings. 



0..1..2..3..4..5..6..7..8..9..10 Be process oriented. 

0..1..2..3..4..5..6.-7..8..9..10 Be open to change and willing to initiate change. 



0..1..2..3..4..5..6..7..8..9..10 Be people oriented. 



0..1..2..3..4..5..6..7..8..9..10 Be future oriented. 



0..1..2..3..4..5..6..7..8..9..10 Consider global perspectives in technology education. 



0..1..2..3..4..5..6..7..8..9..10 Use a vocabulary that reflects the concepts of 

technology education. 



0..1..2..3..4..5..6..7..8..9..10 Have a personal professional development plan. 



Oraduatet of the Technology Tetcher Education Progrun 

0..1..2..3..4..5..6..7..8..9..10 T6ach caicepts and use teaching techniques that are 

technology based. 

0..1..2..3..4..5..6..7..8..9..10 Employ a philosophy which rettects a technological 

base. 

0..1..2..3..4..5..6..7..8..9..10 Recommend the purchase of appropriate equipment 
0..1..2..3..4..5..6..7..8..9..10 Implement technology based activities. 
0..1..2,.3..4,.5..6..7..8..9..10 Use group activities in their instruction. 
0..1..2..3..4..5..6..7..8..9..10 Encourage interdiscipUnaiy approaches. 
0..1..2..3..4..5..6..7..8..9..10 Receive preferred status from prospective employers. 

Other Criteria tPlease Indicate the category) 

0..1..2..3..4..5..6..7..8..9..10 

0,.1..2..3..4..5..6..7..8..9..10 
0..1..2..3..4..5..6..7..8..9..10 
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PROCEDURES 

Please rate the Importance of each procedure by circling a number on the 0 to 10 
scale. (You may ecnt the procedxure ii you wish.) A rating of 0 is a 
recommendation that the procedure be dropped. A rating of 10 Indicates that the 
procedure Is absolutely vital to the assessment of the effectiveness of the change to 
technology teacher education. 

If a procedure should be added, please write It in the space provided - and late 

It 



0..1..2..3..4..5..6..7..8..9..10 Use paper and pendl tests to determine whether the 

students have learned the conceptual structure, 
principles, and litems of technology education. 

0..1..2..3..4..5..6..7..8..9..10 Obsenre seniors to assess their competence in teaching 

technology education. 

0..1..2..3..4..5..6..7..8..9..10 Develop an exit examination to test outcomes of 

technology education teacher education. 

0..1..2..3..4..5..6..7..8..9..10 Verify compliance with nEA/CTIE/NCATE guidelines. 

0..1..2..3..4..5..6..7..8..3..10 Develop and validate national standards for technology 

teacher education. 

0..1..2..3..4..5..6..7..8..9..10 Determine whether the number of students in the 

program represents a viable part of the departments 
program offerings. 

0..1..2..3..4..5..6..7..8..9..10 Analyze the courses required in the program, the 

content contained in each of the courses, teaching 
strategies and methods, assignments, tests, and student 
field ejqperience to determine if they reflect technology 
education. 

0..1..2..3..4..5..6..7..8..9..10 Interview students to assess their belief in life-long 

learning. 

0..1..2..3..4..5..6..7..8..9..10 Interview students to assess the degree to which they 

are people oriented. 

0..1..2..3..4..5..6..7..8..9..10 Verify records of professional development activities of 

faculty 



33 



Prooedures • Continued 
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0..1..2..3..4..5..6..7..8..9..10 Review the content and quality of faculty research and 

publications. 

0..1..2..3..4..5..6..7..8..9..10 EJcamlne assignments and laboratoiy acttvitiet to 

determine the extent of problem solving, concrat 
applications, and tool, madilne and equipment 
utilization. 

0..1.-2..3..4..5..6..7..8..9..10 Examine assignments and laboratoiy activities to 

determine whether th^ reflect cunent education theory, 
current technology, and a future orientation. 

0..1..2..3..4..5..6..7..8..9..10 Obtain data regarding hlrlxig of graduates because of 

their technolo^ education uiomedge from program 
fiaiculty who act as hiring contacts and/or tmtverslty 
placement offices. 

0..1..2..3..4..5..6..7..8..9..10 Poll public school administrators regarding their hiring 

preference for graduates of the program. 

0..1..2..3..4..5..6..7..8..9..10 IJurlng on-site inspections, verify that facility changes 

reflect technology education requirements. 

0..1..2..3..4..5..6..7..8..9..10 Poll students and colleagues within the Institution and 

outside of the Institution to see If the program is 
regarded as technology education. 

0..1..2..3..4..5..6..7..8..9..10 Listen In the halls to learn whether the students and 

fiaiculty speak a language indicative of an understanding 
of the change to tecnnology education. 

0..1..2..3..4..5..6..7..8..9..10 Examine course descriptions to determine whether thqr 

reflect the terminology of technology education. 

0..1..2..3..4..5..6..7..8..9..10 Observe student activities, projects, and products to 

determine if they reflect tecnnology education. 

0..1..2..3..4..5..6..7..r...9..10 Survey students to assess their understanding of the 

social-cultural aspects of technology. 

0..1..2..3..4..5..6..7..8..9..10 Determine whether feculty members are reading and 

talking about technologyr. 

0..1..2..3..4..5..6..7..8..9..10 Determine the level of student and fiaculty enthusiasm 

about the technologyr teacher education curriculum. 



<1 / 
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ProcedorM - Continued 

0..1..2..3..4..S..6..7..8..9..10 Review the adequacy of &culty and staff allocations to 

the program. 

0..1..2..3..4..S..6..7..8..9..10 Examine the currtcuhmi to detennine If the phllosopt^. 

definition, mlssloa statement goals and oblectlves. 
course content and teaming experience reflect 
technology education. 

0..1..2..3..4..S..6..7..8..9..10 Follow up the graduates to determine what concepts 

and actMUes mey are teaching, and the teaching 
techniques they are using. 

0..1..2..3..4..S..6..7..8..9..10 Query recent graduates for their opinions on the 

phllosoidiy and purpose of technology education. 

0..1..2..3..4..5..6..7..8..9..10 Conduct a self study of the program In conjunction with 

an external review. 

0..1..2..3..4..5..6..7..8..9..10 Examine trends In institutional financial support of 

tedmolog/ teacher education. 

0..1..2..3..4..5..6..7..8..9..10 Review reading lists of books and periodicals provided 

to students by faculty. 

0..1..2..3..4..5..6..7..8..9..10 Examine enrollment trends. 

0..1..2..3..4..5..6..7..8..9. J Review faculty qualifications and inservlce development 

efforts. 

0..1..2..3..4..5..6..7..8..9..i0 Evaluate the pacing and time fiame of the change to 

technology teacher education. 

Other Procedures 

0..1..2..3..4..5..6..7..8..9..10 
0.. 1..2..3..4..5..6..7..8..9.. 10 
0..1..2..3..4..5..6..7..8..9..10 



Comments: 
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Appendix B 

Ranked UMttog of CrlttrU and Procedure Stttemente 
Sorted Iqr Qnetttoimalro Cat^ory. 

Mean * Technolo^ Teacher Education Program 

9.55 Laboratoiy Instruction provides oiq;>ortunities for students to reinforce 
abstract concepts with concrete experiences. 

9.50 Instructional strategies emphasize conceptual understanding and problem 
solving. 

9.23 Professional studies component emphasizes the study of technologjr, 
including social-cultural affects and consequences. 

9.22 Laboratories facilitate the learning of broad based technological concepts. 

9.22 Instruction incorporates current technological activities. 

9.17 Philosophy, mission statement, goals and curriculum emphasize 
technological skills as opposed to technical skills. 

9.17 Soclal*cultural impacts of technology are emphasized. 

9. 12 Field experiences are technology centered. 

9.05 Problem solving and decision making abilities are emphasized. 

9.00 Curricula are based on recent research findings. 

8.89 Program Is recognized as technology teacher education by faculty within tlie 
institution, colleagues in other Institutions, and public school 
administrators. 

8.89 General education requirements In mathematics, science, computer science, 
and humanities provide the necessaiy breadth and depth. 

8.78 Library acquisitions reflect contemporary technology. 

8.67 Administrators in the institution agree with and support the philosophical 
change to technology teacher education. 

8.67 Financial support for technology teacher education Is adequate. 

8.67 Courses cover the spectrum of technology education. 

8.50 Textbooks and instructional media reflect contemporary technology. 

8.29 Technology as a body of knowledge, determines program goals and 
structure. 

8.28 Cutting edge technologies are included in insunctional offerings. 
8.22 Facilities represent contemporary technologies. 

8.22 Instruction emphasizes the safe and efflcient use of tools, machines, and 
equipment. 



ERIC 



39 



36 

Tvchnoloff teacher education program - Continued 

7.89 Curriculum Is In compliance wl' a nEA/CTTO/NCATC guidelines. 
7.78 Content Is organized aroiuid technological ^tems. 
6.39 Ttecher preparation Is the primary focus. 

Faculty Members 

9.50 Display a positive attitude toward the technology teacher education 
curriculum. 

9.22 Participate in planned professional development activities to update their 
knowledge and skills. 

9 05 Communicate their understanding of the meaning and topUcatlons of 
teSSgTeducatlon both within and outside the classroom. 

7.78 Are actively engaged In research In technology education. 

7.78 Publish In the field of technology education. 

7.72 Present at national regional, and local conferences. 

Students Are Expected To: 

9.78 Be people oriented. 

9.44 Be future oriented. 

9.39 Demonstrate the ability to teach problem solving techniques. 

9.33 Effectively plan and implement technology education In grades 5-12. 

9.28 Develop and implement curriculum material that reflect a broad 
technological system area. 

9.28 Demonstrate an awareness of societies reliance on technological systems. 

9.22 Plan and Implement teaching-learning activities. 

9.17 Use a vocabulary that reflects the concepts of technology education. 

9.11 Apply current instructional theory. 

9.06 Formulate appropriate objectives. 

9.05 Be open to change and willing to initiate change. 

9.05 Consider global perspectives In technology education. 

9.00 Demonstrate a basic underetanding of tools. ^^^^^.^^^^^^ 
their appUcations In manufacturing, construction, communication, ana 

transportation. 
8.94 Demonstrate knowledge of current technology. 
8.83 Develop conceptual understanding?. 
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students - Continued 

8.78 Develop a curriculum that anafyzcs the social-culturel affects of technology. 
8.55 Design evaluation devices and instruments. 
8.50 Be process oriented. 

8.39 Have a personal professional development plan, 

8.00 Develop a curriculum that illustrates how new technology is created. 

7.67 Develop or use a new or existing taxonomy. 

7.50 Identify the concepts, principles, and systems of technology. 

Panelist Write-ins: 

0.55 Revise existing courses and develop new courses to reflect contemporary 
technology education content and methods. 

0.55 Use group activities and individual activities in their instruction, 

0.55 Implement new and revised courses. 

Graduates of the Technology Teacher Education Program: 

9.78 Employ a philosophy which reflects a technological base. 

9.61 Teach concepts and use teaching techniques that are technology based. 

8.83 Implement technology based activldes. 

8.78 Use group activities in their instruction. 

8.72 Encourage Jnterdlsclplinaiy approaches. 

8.10 Recommend the purchase of appropriate equipment. 

6.67 Receive preferred status from prospective employers. 

Procedure Statements 

9.50 Examine C:e curriculum to determine if the philosophy, definition, mission 
statement, goals and objectives, course content, and learning experience 
reflect technology education. 

9.22 Analyze the courses required in the program, the content contained in each 
of the courses, teaching strategies and methods, assignments, tests, and 
student field experience to determine if they reflect technology education. 

8.72 Validate the impact of irEA/CTTE/NCATE national standards for technology 
teacher education upon the change process. 

8.50 Verify compliance with ITLA/CTTE/NCATE guidelines. 

8.44 Follow up the graduates to determine what concepts and activities they are 
teaching, and me teaching techniques they are using. 



Proceduret * Continued 

8.39 Observe student activities, projects, and products to determine if they reflect 
technology education. 

8.33 Examine assignments and laboratory activities to determine whether they 
reflect current education theory, current technology, and a future 
orientation. 

8.33 Observe students In the final year of the program to assess their 
competence in teaching technology education. 

8.22 Query recent graduates for their op nions on the philosophy and purpose of 
technology education. 

8. 11 Develop an exit examination to test outcomes of technology education 
teacher education. 

8.05 Survey students to assess their understanding of the social- cultural aspects 
of technology. 

8.00 Conduct a self study In conjunction with an external review. 

7.89 Review faculty qualifications and Inservice development efibrts. 

7.83 Examine assignments and laboratory activities to determine the extent of 
pioblem solving, concept applications, and tool, macliine and equipment 
utflizatlon. 

7.44 Review reading lists of books, periodicals, magazines, newspapers provided 
to students by faculty. 

7.44 Examine course descriptions to deterritoe whether they reflect the 
terminology of technology education. 

7.33 Examine trends in institutional financial support of technology teacher 
education. 

7.28 During on-site Inspections, verify that facility changes reflect technology 
education requirements. 

7.22 Students and faculty are enthusiastic about the technology education 
curriculum. 

7.16 Evaluate the pacing and time frame of the change to technology teacher 
education. 

7. 1 1 Determine whether faculty members are reading and talking about 
technology. 

7.05 Verify records of professional development activities of faculty. 

6.89 Obtain data regarding hiring of graduates because of their technology 
education. 

6.78 Review the adequacy of faculty and staff allocations to the program. 
6.78 Verify faculty research and writing. 
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Procedures - Continued 

6.61 E» : nc enrollment trends and recruitment procedures. 

6.44 Interview students to assess the degree to which they are people oriented. 

6.28. Use paper and pencil tests to determine whether the students have learned 
the conceptual structure, principles, and ^tems of technology education. 

6.05 Poll public school administrators regarding the hiring preference for program 
graduates. 

6.05 Interview students to assess their belief in life-long learning. 

6.00 Poll students and colleagues within the institution and outside of the 
institution to see If the program is regarded as technology education. 

4.39 Listen in the halls to learn whether the students and faculty speak a 
language indicative of an understanding of the change to technology 
education. 

Panelist Write-ins: 

0.55 Review portfolios of student work. 

♦ Rated for Importance on a 0 to 10 scale with 0 meaning delete from 
procedure list and 10 meaning absolutely essential. 
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^pendlx F 

Pinel Responses Comjiared with NCATB-Gnidellnes and Evidence of Compliance 

The table below lists the criteria suggested by the panel of experts in this study and 
the NCATB curriculum guldellne(s) most doser^ assodated with the respective suggested 
criteria. Ihe criteria are presented in the order that tht^ appeared on the Roundu 
questionnaire and are not ranked in any wi^. Beside each criterion or set of criteria, 

Srocedures suggested by th^ pand and evidence requested to establish compliance with the 
[GATE 'nuTlculiim guidelines are noted. 

For Clarity: 

The criteria and procedures identified by the pand in this research are presented tn italics. 

The NCATE-gulddincs and corresponding evidence of compliance are presented in normal 
typeface. 



Technology Teacher Education Program 
Criter{a/NCATE-Guideline(s) Pracetf "^ii^s/Evidence of Compliance 



Courses provide a balanced coverage of the 
spectrum of technology education. 

NCATE «3.2 Instructional content is 
drawn from the following content 
organizers, 3.2.1 communication. 3.2.2 
construction. 3.2.3 manufacturing, and 
3.2.4 transportation. 



General education requirements in such 
areas as mathematics, science, couypuler 
science, social sciences, and humanities 
provide the necessary breadth and depth. 

NCATE #2.0 Courses in math, science, 
and rdated areas in the general education 
component provide the necessary depth 
and oreadth for technology education. 



Examine course descriptions to detemUne 
xvhether they reflect the terminology of 
technology education. 

3*2*1,*2,«3,«4 For each of the four 
tedmology areas there is evidence of 
specific, required and well devdoped 
academic and laboratbiy courses and 
experiences that: 

- provide knowledge and understanding of 
technological ^tem. concepts, and 
tedmicaimeans. 

- devdop within each student the ability to 

Krform in the several technical areas of 
e technological system using state-of-the- 
art instruments, devices, equipment and 
materials. 



No suggested procedure. 



2.1 - Mathematics courses required of all 
students and taught by faculty fully 
qualified in mathematics. Courses are: (a) 
area appropriate, and (b) of the depth and 
quality to provide a solid foundation for 
continued mathematical devdopment In 
addition, the prindples of mathematics are 
incorporated into all technology courses. 

2.2 Same as 2.1 plus: Concepts and 
prindples of the sdences are utilized in 
appropriate and meaningful applications in 
tne analysis of technical problems and in 
the design of technological devices and 
systems. 

2.3 Rdated areas of study (General 
Education) are a part of the required 
technology teacher education program. 
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Criter(a/NCATE-Gaidellne(t) 

Prdbtem sobtng and decision making 
QbilSties are emphoslzecL 



NCATE # 3.7 The program develops 
student problem-solving and decision 
making abilities involvmg human and 
material resources, processes and 
technological systems. 



Proceduref /Bridence of Compliance 

Jxannine assignments and laboratory 
actbtties to aetermine the extent of problem 
soivtng, concept appUcatioris, and tool 
machSie and equq)ment utHizatlon. 

3.7 document student work and written 
and other evaluations. 



Labmdory instruction provides q;>portunities 
for students to reiT\force tedmoiogical 
concepts with concrete experiences. 

NCATB 3.8 The program provides activity 
oriented laboratory instruction with 
student reinforcing abstract concepts with 
concrete experiences. 



No suggested procedures. 



3,8 Course syllabi and photo records, 
video, or other graphic documentation, that 
students are involved in specific laboratory 
instruction that focuses on stated abstract 
concepts and that the program provides for 
the reinforcement of abstract concepts with 
concrete ejqpcrlences. 



Instruction emphasizes the safe and 
eOkient use of a wide variety of tools, 
machines, and equipment. 

NCATB 3.5 The program assists students 
to apply tools, materials, machines, 
processes, and technical concepts, safely 
and eflldently. 



No suggested procedure. 



3.5 Logs of safety lessons taught and 
monitoring of student performance. 



Curri adum is tn compliance with 
rrEA/CTTE/ NCATE guidelines. 



Verify compliance with ITEA/ CITE/ NCATE 
guiaelines. 



Philosophy, mission statement goals and 
cuTTtemmi emphasize technological skills as 
q[>posed to technical skills. 



NCATB 1.1 The technology education 
program is based on a sound mission 
statement with stated goals and objectives 
which reflect the intent of technology 
education. 



Examine the curricuijum documents to 
determine tf the philosophy, definition, 
nUsston statement, QoaLs and otifectives, 
course content, oridL learning experience 
reflect technology education. 

1.1 Written mission statement, goals and 
objectives, that are compatible with: 

(a) contemporary philosophy, practice, 
and current research flndings for 
curriculum design. 

AND 

(b) the definition and description of 
technology education as published by 
ITEA. 



4-) 
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Crfterla/NCATB-Giiideliiie(s) 

Instwctlon tncofporates current technolDalcal 
QcttviUes. cutttng edge technologfes. and 
provides for the ejqHoration of new and 
emerging technologies. 

NCATB 3.11 The program develops In 
student the abllily to apply technological 
knowledge and skills to understanding 
various past^present-future technology 
systems. 



Pn>Mdum/B¥ldenM of CompUance 

No suggested procedure. 



3.11 Formal evaluation results that 
students develop positive and appropriate 
attitudes, and knowledge and skills in the 
use and analysis of information concerning 
the composition and operation of various 
past, present, and future technological 
systems. 



Professional studies comp<ment emphasizes 
the study technology, including social- 
cultural effects and conseqjieru:es. 

NCATB 3.1 The program is based on 
fundamental knowledge about the 
development of technology, its effect on 
people, the environment and culture: and 
industiy. its organization, personnel 
systems, techniques, resources and 
products and their socio-cultural Impacts. 



Survey students to assess their 
understanding of the social-cultural aspects 
of technology. 

3.1 Ihe core program includes academic 
courses and experiences, taught by 
qualified faculty, that {Hfovlde fundamental 
knowledge and understanding of: 

- the hiSoiy and evolution of technology 
and technological systems including 
inventions, innovations, developments, 
people, events, and idaces. 

- the study of technological development 
and its effect on people, cultures, and the 
environment. 

- industry, its organization, personnel 
systems, tedmiques. resources, and 
potKlucts. 

- industry and its social-cultural impact. 



Ubrap' acquisitions rejlect contemporary 
technology. 

Textbooljs and instructional media reflect 
contemporary technology. 



No suggested procedure. 

Review reading lists of books, periodicals, 
magazines. r\ewspapers provided to 
stments by faculty. 



Professional fidd experiences are 
technology education. 

NCATB 6.0 The curriculum includes a full- 
time student teaching experience 
conducted in a technology education 
^roflam under the supervision of program 
acjity and a master teacher in the school 
setting. 



f 



No suggested procedure. 

6.0 Faculty vltae. faculty assignments, 
published policies, program requirements, 
master teacher qualifications, and 
technology education faculty qualifications, 
and transcripts of graduates, that each 
person has participated in a well structure, 
carefully monitored and supervised fiulli 
time student tearhtnf eyoerience. The 
program is administered and supervised by 
gushed tedinology education program 
faculty in cooperation with a certified 
master teacher in an approved technology 
education program laboratory and 
dassro setting. 
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Crlter(a/NCATE<Siiidellne(t) 
NCATB 6.0 continued. 



ProceduiYs/Eridence of Compliance 

6.1 Student teaching assignments, weekly 
student teacher activity plans, reports from 
master teacher, on site visitation reports by 
supervisors, orientation programs and 
student teaching plans (a) In a public or 
private sdiod setting, (b) a wide range of 
experiences were prodded. 

6.2 Stud<:nt had full responsibility for the 
preparation, teaching and management of 
classes and laboratories. 



Content Is organized aroimd technologiccd 
systems, 

NCATB 1.2 The program is based on an 
organized set of concepts, processes and 
systems that are urL4udy technological. 



No suggested procedure. 

1.2 Evidence In course syllabi of a focus 
on: (a) spedflc technological concepts, 
principles and Intellectual processes 
(b) relationship to other technological 
and social systems. 



Technology, as a body of knowledge, 
determines program goals and sbvcture. 



NCATB 1.3 The study of technology Is 
reflected in cuirlcular design, course 
outlines, instructional strategies, and 
evaluation of student work. 



Analyze the courses required In the 
program the content contained In each of 
the courses, teaching strategies and 
rryethods, assignments, tests, and student 
field experience to determine they refiect 
technology education. 

1.3 EMdence that curriculum documents 
are compatible with: 

(a) brochures, admission oollcles, 
department activities such as 
conferences, minutes of meetings, etc. 

(b) course outlines 

(c) Instructional strategies 

(d) student evaluation 



Instructional strategies emphasize 
conceptual understanding and problem 
soh)tng. 



No suggested procedure. 



Curricula ar^ based on recent research 
findings. 



Examine asstanments and laboratory 
activities to determine whether they refiect 
current education theory, current 
technology, and a fixture orientation. 



NCATE 1,0 The technology education 
program provides a curriculum that Is 
consistent, with current research findings 
for curriculum design. 



1.1 Written mission statement, goals and 
objectives, that are compatible with: 

(a) contemporary philosophy, practice, 
and current research findings for 
cuiTiculum design. 

(b) the definition and description of 
technology education as published by 
ITEA. 
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Criter(a/NCATB-Giiideline(s) 

Facilities represent contemporary 
technologies and JocQState the leomlng of 
broad based technological concepts. 



Administrators in the institution agree with 
and support the philosophical chcmge to 
technology teacher education. 



Ftnancial support for technology teacher 
ediication is adeq^iate. 



Program is recognized as technology teacher 
edsIcaHon by faculty within the institution. 
coUeagues in other institutions, and public 
school administrators. 



ProoedunM/Bvidence of Compllmnce 

During on-site mspecttons. ver^ that 
facility changes reflect tedvidlogy education 
requirements. 



Review the adequacy of faculty and sto/f 
allocations to the program. 



Examine trends in iristttutional financial 
support of technology teacher education. 



Pod students and colleagues within the 
Institution and outside of the Institution to 
see if the proaram is regarded as 
technology ecwcation. 



Faculty Members 



Participate in planned professional 
developmeni activities to update their 
technological hiowledge and skills. 



Review faculty qualifications and inservice 
development efforts. 

Verify records of professional development 
activates offaamy. 



Are actively engaged in research in 
technology education. 

Publish in the field of technology education 

Present at rwtional regional and local 
conferences. 



V^fij faculty research and writing. 



Displau a positive attitude toward the 
technology teacher education curricuLum 



No suggested procedure. 



Communicate (heir understanding of the 
meaning and tmpUcations of tecfmology 
education both within and outside the 
classroom. 



Determine whether faculty members are 
reading and talking about technology. 



Student 

Identify the concepts, principles, and 
systems of technology. 



NCATB 3.4 The program assists students 
in developing insijght. understanding, and 
application of technological concepts, 
processes, and systems. 



Outcomes 

Use paper and pencil tests to determine 
whether the students have learned the 
conceptual structure, p)1nciples, and 
systems of technology education 

3.4 Evidence from course sj^abi, 
instructional design and meChodology, and 
from results on evaluation instruments to 
measure ... (a) attaining a high level of 
insist and understanding and (b) 
dcvdoping the ability to apply basic 
technological concepts. 
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Crlteiia/NCATB-GuidellneM 

Effectively plan and bryplement technology 
^tucatlon in grades K - 12. 

NCATB 4,1 Develop a stratefi^c plan that 
includes a mission statement, rationale for 
change, goals and objectives, action steps, 
as wdl as program evaluation strategy. 



Fmocedures/Bvidence of CompUanoe 

No suggested procedure. 

4.1 Documented evidence that senior level 
stur'^nts attain knowledge and skills that 
enable them to develop strategic plans for 
technology education programs that 
include mission statemenb, rationale for 
change to technology education, 
appropriate goals and objectives related to 
the mission statement, action steps for 
implementing the plan and a program 
evsuuation soategy. 



Apply current instructional theory. 

Develop or use a new or existing taxonomy. 

Plan and Implement teaching leaming 
activities. 

Design appropriate evaluation devices and 
instruments. 

Develop and implement curriculum material 
that refect a broad technological area. 

NCATB 4.6 Develop lesson plans, organize 
materials and present psychomotor, 
affective, and cognitive instruction. 



Fbrmulote appropriate objectives. 

NCATB 4.2 Select content based on goals 
and objectives within the four content 
organizers. 



Observe students in the final year of the 
program to assess their competence in 
teaching techntAogy education. 

Develop an exit examination to test 
outcomes of technology education teacher 
education. 



4.6 Documented evidence In the form of • 
photo records, samples of written student 
work, and faculty ^raluations of student 
p^ormance that students are capable of: 

(a) developing wdl structured quality 
lesson plans . . . 

(b) selecting appropriate instructional 
materials . . . 

(c, d, e) designing, developing, presenting, 
and evaluat£g instruction ... in the 
psychomotor realm, affective realm, and 
cognitive realm . . . 



No suggested procedure. 

4.2 A random sample of senior level 
students in each of the four content 
categories be able to develop a program 
that will (a) use the approivlate content, 
(b) be of professional quality and contain 
as a minimum, 1) a mission statement, 2) 
a rationale for change, 3) goals, 4) 
objectives, 5) action steps Tor 
implementation, and 6) a program 
evaluation strategy. 
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Criterfo/NCATB^uldellneM 



Demonstrate a basic understanding of tcdls, 
wachSnes and process and thetr 
applications in manitfacturing, constructioru 
contmuniccdion. and transportation 

NCATB 3.3 The levd and scope of skills in 
the use of tools, instruments, and 
machines to be identified and incorporated 
into the programs. 



Pnocedum/Brldeuce of ComplUnce 

ReDiew por{foUos of student work. 

Observe student actbtties, prqfects, and 
products to determtne if (hey reflect 
technology education 

3.3 (See compliance for 3.2.1-4 b.) 
Specific, required and yurill developed 
academic and laboratoiy courses and 
experiences ... [in each of the four 
content categories] that: 
(b) develop within each student the ability 
to perform in several technical areas . . . 



Develop a curnculum (hat illustrates how 
new technology is created 

Develop a cwriculum that anaJuzes (he 
social-cultural affects of technology. 

Demonstrate (he ability to teach problem 
solving technicpjies. 

NCATB 4.3 Structure an educational 
environment in the classroom and 
laboratory to accommodate the 
instructional process. 



JVd suggested procedure. 



4.3 Students are involved in structuring an 
educational environment in a classroom 
and laboratoiy that is compatible ivith the 
requirements of the instructional processes 
required for a given field of technology. • 



Demonstrate knowledge of current 
technology. 

Demonstrate an awareness of societies' 
reliance on technological systems. 

Develop conceptual understandings in 
technology ana technology education 

Be process oriented 

Be open to change and wiRing to tnitiate 
change. 

Be people oriented 

Be future oriented 

NCATE 3.10 The program develops 
students* attitudes, knowledge, and skills 
regarding how technological systems 
function. 



Interview students to assess (he degree to 
wNch (hey are people oriented 

Interview students to assess (heir belief tn 
l^e-long learning. 

Usten in (he halls to learn whether (he 
students arvdfacultu speak a larmjoge 
indicative of an understanding of the 
diange to technology education. 



3.10 Evidence fi'om evaluation and 
assessment instruments, samples of 
student work, documented Instructional 
content and other activities of the level of 
knowl^ge and the type and level of skills 
attained oy students regarding how 
technological systems function. 
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Pnoceduivs/Bvideiice of Compliance 

No suggested procedure. 

B.3 Documented evidence from course 
syllabi, faculty lesson plans, textbooks. 
Instructional materials, and evaluation 
Instruments that all students are 
knowledgeable about various cultures and 
their past. They are able to discuss 
current contributions to the evolution of 
technical means and technical systems and 
the interrelation between and among 
cultures and nations with respect to the 
creation and use of technical means and 
adaptive systems. Ihey imderstand the 
rdaUon or technical means and systems to 
human society, other life forms and the 
environment. 



Usten in the haUs to team whether the 
students and facultu speak a buiguage 
indtcattoe of an understanding of the 
change to technology education. 

Interview stidents to assess their belief in 
life-long learning. 

4.12 Prepared and recorded written • 
material that each student . . . has a plan 
de^0ied to provide direction for continued 
personal ana professional growth. 



Criteria/NCATI^Guideliiie(s) 

Consider global perspectiues in technology 
education. 

NCATE 8.3 Ihe students are Introduced to 
multicultural and global perspectives as 
they relate to the study of technology. 



Use a new vocabulary that rejlects the 
concepts of technology education. 



Have a personal professional development 
plan. 

NCATE 4.12 Establish a professional 
development plan for continued personal 
and professional growth. 



Graduates of the Technology Teacher Education Program 

Teach concepts and use teaching techniques 
that are technology based. 

Reconrmsnd the purchase of appropriate 
equipment 

Implement technohgu based 
activmesI'oUow up the graduates to 
determine what concepts and activities they 
are teaching, and the teaching techniques 
they ore using. 



Enwloy a philosophy which reflects a 
techxological base. 



Query recent graduates for their opinions 
on the phUosophy and purpose of 
technology education. 



Use group activities in their instruction. 

NCATE 4.4 Select appropriate instructional 
strategies for individual and group 
instruction, including safety Instruction. 



Afo suggested procedure. 

4.4 Documented evidence, in the form of 
student work, that students are capable of 
designing and selecting appropriate 
Instructional strategies for group or 
individual instruction . . . 
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Criterio/NCATB-GuldeUneM 

Encourage MerdisctpUnary approaches. 

NCATB 3.12 Ihe program develops In 
students an understanding of the 
application of other areas of knowledge 
(math, science, history, etc.) to tedmologjr 
and the solution of human and social 
problems, including appropriate skill in use 
of tools and machine^) 



Receive preferred status as technology 
education teachers from prospective 
employers. 



Pnocedures/BHdence of Compliance 

No suggested procedure. 

3.12 Documented evidence In the form of 
course qilabl. photo records, student 
woric results of student evaluation and 
other data that students are able to utilize 
knowledge, information, and skills from 
other dls^;)llnes In the analysis of 
proposed tedmologlcal solutions to htmian 
ana social problems 



PoU public school admtnistrators regardtng 
the mrtng preference for program graduates. 

Obtain data regarding hiring of graduates 
because qf their tecmblogy eauoatlon. 



Other Suggested Procedures 

Validate the Impact of TTEA/CTTEINCATE national standards for technology teacher education 
upon the change process. 

Conduct a self study in cor^unction with an external review. 

Evaluate the pacing and time frame of the change to technology teacher education. 

Examine enroUment trends and recruibnmt procedures. 
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^pendlz O 
Round m Letter to PineUste and 
Draft TechnoloCr Teacher Education Checklltt 



June 20, 1990 



Dear Panelist: 

Hiank you for continuing to serve as a member of the panel In our project, *The 
Development and P ilot Te sting of a Research Design for Evaluating the Effectiveness 
of Change to nEA/CTTE/NCATE Curriculum Guidelines in Technology Teacher 
Education," which is being conducted under a grant from the Council on Technology 
Teacher Education. 

As you will recall, during the first round, panelists suggested criteria and 
procedures for assessing the effectiveness of program change from industrial arts 
teacher education to teamology teacher education. During the second round, 
panelists evaliaated the ImporSnce of each criterion and each procedure. These 
responses provided the foundation for the preparation of the •Technology Teacher 
Education Cheddlsf The checklist, which attempts to focus upon the most 
fundamental questions in the assessment of a technology teacher education 

Erogram, Is now being sent to panelists for their comments. Field testing, using the 
Iheckllst as an assessment guide, will begin soon. 

Please review the attached Technology Teacher Education Checklist and give me 
your reactions to it by writing your notes directly on the canary copy and I'etuming 
it in the enclosed repty envelope. A copy of the report of the research to date is 
enclosed to provide you with as much background as you may find Interesting or 
helpful 

Thanks again for your continued participation in the research. Your CTTE 
colleagues - and their students - will benefit from your contributions to this effort. 

Sincerely, 



Daniel L. Householder 
Professor and Project Director 
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Draft Technolocr TMcher Educmtion Checklift 



A, Elxamlne the catalog and a sample of curriculum documents to deteraiinc the 
degree to which: 

1. The philosophy, mission statement, and goals and objectives reflect the 
ITEA definition of technology education. 

2. Study Is required In these technological systems: communication, 
construction, manufactming, and transportation. 

3. Courses in mathematics and science are reqtiired. 

4. Full-time student teaching experience in a technology education setting is 
required. 

5. Required reading lists reflect technology education. 



B. Interview the department head to determine the degree to which: 

1. Funding is adequate to support the current technology teacher education 
program. 

2. Faculty and stafT allocations are adequate to serve student enrollments in 
technology teacher education. 

3. The written departmental plan for faculty professional development and 
technological updating is adequate to prepare them for coi::temporary 
technology teacJier education. 

4. Current faculty research activities are directed toward technology 
education. Ask for a sample of current research activities to review, 

5. Enrollment Is adeouate, stable, or increasing relative to enrollments prior to 
the change to technology education. 

6. The written departmental implementation plan for technology teacher 
education addresses the process of organizational change. 



C. Interview faculty and revlejv recent blodata information, faculty publications, 
copies of presentations, and manuscripts being considered for publication to 
verify whether: 

1. Faculty are writing about and giving presentations in technology education. 

2. Current faculty research activities are directed toward technology 
education. 

D. Observe professional classes to determine the degree to which: 

1. Instructional methods reflect an emphasis on problem solving and declsi 
making. 

2. Instructional materials reflect technology education. 
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TTEC Draft - Continued 

E. Inspect laboratoiy fkdllUes to ascertain the degree to which: 

1. Equipment is appropriate for provldlntf students opportunities to 
understand the concepts and practices oi contemporary technologies. Such 
indicators may indude equipment for desk-top publishing, robotics, CNC 
milling. CAD. video production, materials testing, laser applications. 

F. Interview students and examine student logs or other required student work to 
discern whether: 

1. r-ie problem solvii^ process and decision-making rationale are 
incorporated into gracSng. 

2, Environmental consequences and social-cultural effects of technology are 
reflected In student activities. 

G. Interview dean, provost, or president to leam their perceptions of technology 
education. 



H. Listen to conversations and dlscussion^i to discern the degree to which: 

1 . The terminology used by faculty and students reflects technology 
education? 

2. Faculty and students appear to be enthusiastic about technology 
education? 
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Appendix H 

CommenU on the Draft Technology Teacher Education Checklist 



A. Comments: 

"Seems to be worded In the context of what external evaluators would do." 

1. ... objectives of the program reflect the . . . 
Comments: 

1 am sure the re Is a reason, but why emphasis on UEIA?" 
"Doesn't CTTE and other documents explain what TE ls7' 
"[ITEAl What about state definitions?" 

2. . . . communication, biotechnology, construction . . 

. . . required in technological systems such as communication . . . 
Comments: 

"Production, biotechnology?" 
"Checking for its degree of depth." 

"May be configured a different way (I.e., communication, production , 
transportation . . .)." 
"These are changing." 

3. . . . and are prerequisites to a study of technology. 

. . . mathematics, social science, and the language arts are required. 

Courses in mathematics, science and other disciplines are required and 

reinforce technological concepts. 

Comments: 

"Computer science?" 

"At least 2 courses in each area." 

4. Pre-student teaching experience . . . 
Comments: 

important that [the technology education settlngj Is not lA." 
'The student teaching must have a technology ec* ication component. If we 
place student teachers in a traditional lA prograi . they may not develop into 
a TE teacher without some TE component of their student teaching." 

5. . . . reflect technology and technology education. 
Comments: 

"(Reading lists] in professional courses? Or in technical courses as weU?" 
SUGGESTED ADDITIONS: 

"Learning activities or experiences are representative of technology education and 

reflect objectives of course(s)." 

*The impacts of technology are studied." 

"Social-cultural elements are included in addition to technical content." 



B. Comments: 

"Seems to be worded in the context of what external evaluators would do." 
"Also, annual faculty reports should be available." 

'TJeeds to be worded to reflect the change to technology teacher education." 
h Comments: 

"How Is this different from our industrial arts program?" 

2. Comments: 

"How Is this different from our Industrial arts program?" 
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TTEC Draft Comments - Continued 

3. Comments: 

•Good." 

^' TSStaglhe various stages of development this will vary a ^eat deal-In fkct. 
it may decline for several years during the change process. 
"Good nvraiber of majors?* 

5. Comments: 

"Good." 

*Ts this necessary?' 

"Contintdng curriculum improvement? 

SUGGESTED ADDITIONS: 

"Faculty are committed to the phUosophy and objectives of technology education." 
"Philosophical support is provided by the administration and liaculty. 
"Hie program has changed to a TE philosophy." 

C. . . . for publication and service activities to verify whether: 

. in technology and technology education. . „ 

. writing end giving presentations about technology education. 
. writing schoEirly papers about ... 



2. 



. technology education topics of issues. 
. faculty research and service activities . . 
Comments: 
•Good." 



SUGGESTED ADDITIONS: 

"Involvement in professional associations." 

"Content reflects the philosophy of technology educators. 



D. . . . professional and technical classes . . . 
Comments: 
"Also lab classes." 

1. . . . emphasis on tecnnologlcal problem solving 
. . . problem solving and a process orientation. 
Comments: 

"Difiicult, but of sor^.e limited value." 



2. . . reflect major elements of technology education (e.g. systems, 
environmental and societal Imracts. use of technological devices, etc.). 
Instructional methods reflect a study of technology. 

l?™*TKy should reflect contemporaiy tCChhQlQgY- How might these be 
different from materials that reflect indusWal arts or industrial technology? 
Are faculty continually developing and updating instructional materials to 
reflect changing technologies?" 



D 4 



TTEC Draft CommenU • Continued 
SUGGESTED ADDITIONS: 

"Professional involvement of faculty and students In campus, state/province, 
national and international organizations that promote technical teaoiing." 

E. . . . equipment for computers, desk-top publishing . . . laser applications, 
conveyers, wind tunnels, bar coding, '"white collar space", processing tools and 
machines. 
Comments: 

"What about facilities to develop curriculum materials?' 

"Biotech?" 

"Simulators," 

"[CNC miUingl table-top or full-sized?" 
" Critical ." 

SUGGESTED ADDITIONS: 

Teaching strategies used in classes." 

•The fecmties have been designed to support conceptual teaching, including 
individual and group problem-solving." 



F. , . . student logs and assignments given by faculty . . . 

L Comments: 

"Good." 

"What are the results of student activities that represent their class work 

(projects vs. understanding? How are the students evaluated in their lab 

classes?" 

"Of some value," 

2, Comments: 

"Good." 

SUGGESTED ADDITIONS: 

'Technical content is supportive of the technology thrust." 

"Learning experiences, assignments, etc. are appropriate to technology education." 
'Technology is understood as a discipline." 

"The elements of technology education are understood and Integrated into their 
total philosophy of education." 

'The systems and concepts of technology are studied." 

"A TECA chapter exists and is acUve in supporting the TE curriculum." 



G. . . . their perceptions and support of . . . 
. . . technology education, present and future. 
Comments: 

"For what purpose? Once we know their perceptions, liien what?" 
"How about department heads in related departments within the college?" 
"Limited value, depending upon organization." 



H. . . . discussions and observe student activity to . . . 
Comments: 

"Follow-up of graduates?" 
Hard to efiecuvely measure." 



i 
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TTBC Draft Comroento • Continued 

1. ... reflects technology and technology education. 

2. Faculty and students appear . . . 
CommenU: 

"Good." 

SUGGESTED ADDITIONS: 

"Projects and activities are oriented toward the study of technology." 

I. SUGGESTED ADDITIONS: 

Review library holdings to ascertain if the holdinte are adequate in Technology 
Education to support student and faculty research. 

J. SUGGESTED ADDITIONS: 

Interview consumers (i.e.. principals) to ascertain if the graduates c£in implement 
technology education." 

GENERAL COMMENTS 

"Should you have anything pertaining to a person with a non-tradlUcnal backgroimd 
coming into the field since tJiat very Ukely will happen more in the future?" 

To be used in 'self -assessment by dept. faculty? Also to be used for external review 
1-2 years prior to NCATE?" 

'Tnis looks excellent." 

"Would it be appropriate to interview or visit critic teachers in technology education 
to determine whether they are In fact teaching technology education? Isuspect this 
might also be ascertained in [item] F above." 

"Good checklist." 

"Do we have an adequate research base to now say that technology education is 
significantly better? I would appreciate copies of such research." 

"Criteria do not seem to reflect the items on the checklist." 



